How Many Labellers Revisited – Naı̈ves, Experts, and Real Experts

A database of non-native German productions was annotated by three different groups: by experts using detailed, localised labels as well as coarse, global labels, and by phoneticians and naı̈ve subjects, using the same coarse global labels. For the detailed annotation, segmental and supra-segmental labels were given segment-based and word-based. The global annotation consisted of a turn-based assessment of intelligibility, nonnative accent, melody, and rhythm. Moreover, we use a large, specialised prosodic feature vector for modelling native vs. nonnative speech. We study relationships between detailed and global labels, analyse the quality of expert and naı̈ve labellers, and present an automatic system for predicting a speaker’s score for the global labels.